1. Field of the Invention
The present invention relates to the field of light field cameras, radiance cameras, directional light capture, 3D cameras and 2D/3D switchable cameras.
2. Prior Art
The advent of nano-scale semiconductors is making it possible to have sufficient computing resources within a typical mobile device, such as a cell phone or a tablet PC, for example, to process high resolution visual information received and/or transmitted through the high data rate mobile networks such devices can typically connect to. A typical mobile device now has a 1M pixel display and an 8M pixel camera allowing the mobile user to view and capture high resolution visual information. With such visual communication capabilities, mobile devices are on the verge of being capable of processing 3D visual information. Although the capture and recording of 2D visual information by ultra compact cameras that meet the stringent volumetric constraints of the mobile devices is now main stream, such is not the case for the capture and recording of 3D visual information. Even in the case of the capture and recording of 2D visual information, the stringent volumetric constraints of the mobile devices is still making it difficult to have cameras with advanced auto focusing capabilities embedded in such devices. The main reason being the bulkiness, poor performance and excessive added cost of incorporating advanced auto focusing features into cameras targeting the mobile devices. A promising prospect for possibly overcoming some of these constraints is a class of cameras known as light field cameras which are capable of capturing information about the directional distribution of the light rays that enter the camera. Besides providing the mobile user with the ability to capture 3D images, the ability to capture the directional information of the light would also enable computational (digital) focusing which would allow the mobile user to capture the entire light field without regard to focusing, then leverage the ample processing capabilities of the mobile device to computationally focus on any desired aspects of the captured light field. In that regard a light field mobile camera would in effect leverage the abundant processing resources now becoming typical in mobile devices to get rid of the expensive and bulky auto focusing. The problem in realizing such a prospect, however, is that the current state-of-the-art light field cameras are inherently bulky in themselves and not at all suited for being embedded in mobile devices. Before proceeding to describe the details of the current invention, the following discussion puts into perspective the current state-of-the-art of light field cameras approaches and their salient characteristics.
Conventional cameras do not record the directional information of the light it captures. A conventional camera captures only a two-dimensional (2D) image that represents a one to one correspondence in light originating from a point in the viewing scene to a corresponding spatial position (pixel) on its photo-detector (PD), as such spatial information is captured but all of the directional information is lost. In contrast to conventional 2D cameras, light field cameras capture both the spatial as well as the directional information of the light. Light field cameras are able to capture both spatial and directional information of the light because they are able to record the radiance of the light, which describes both spatial and directional (angular) information, and is defined as the radiant flux of the incident light per unit of area per unit of solid angle (measured in W.m−2.Sr−1). A light field camera, therefore, is able to sample the four-dimensional (4D) radiance, in so doing captures both the two dimensions of spatial and the two dimensions of directional distributions of the light it captures. Being able to record the radiance, a light field camera therefore captures all of the light field information needed to post-capture focusing, reduce the noise, or change the viewpoint; i.e., three-dimensional (3D) image capture.
FIG. 1A illustrates a prior art light field camera implemented using an array of conventional cameras whereby each of the cameras records an image of the light field from a different perspective. The captured images may then be combined to form the captured light field. The drawbacks of this approach are rather obvious; in order to capture a reasonable angular extent with each camera in the array, the array of objective lenses will span a much larger area than their photo-detectors and will each have a rather large optical track length, thus making the whole camera array of FIG. 1A be limited in terms of the number of views of the light field it can capture, and excessively bulky, thus not at all suitable for embedding in mobile devices.
FIG. 1B illustrates another prior art light field camera implemented using the principal of integral imaging. In this light field camera approach, which is also known as a plenoptic camera, only one objective lens is used and a lenslet or micro lens array is placed near the camera photo-detector to sample the aperture of the camera. The image captured by the plenoptic camera would be made up of an array of sub-aperture images of the light field each recorded by the group of pixels underneath each of the micro lens elements. Each of the sub-aperture images captured by the plenoptic camera would represent a parallax sample of the light field. Although the plenoptic camera of FIG. 1B would potentially provide a higher number of views of the light field and would also be volumetrically smaller than the camera array of FIG. 1A, the increase in the number of views would be at the expense of reduced spatial resolution. In addition, similar to the camera array, for the plenoptic camera to cover a reasonable angular extent, it must employ an as large as possible diameter objective lens which in turn requires a large optical track length, thus make the plenoptic camera also bulky and not at all suitable for embedding in mobile devices.
FIG. 1C illustrates yet another prior art light field camera implemented by using the principal of frequency domain analysis of the light field. In this type of prior art field camera, which although is conceptually equivalent to the plenoptic camera of FIG. 1B, for differentiation will be referred to as radiance camera, is implemented by placing a non-refractive two-dimensional array of pinholes, basically a mask, either in front of the objective lens or in between the main lens assembly and the photo-detector of an otherwise conventional camera. The image captured by such a camera is, therefore, a Fourier domain convolution of the incoming light field with the known non-refractive light field modulation weighting function of the mask. This camera actually captures the 4-D light field directly in the Fourier domain, thus the values recorded by each pixel of the 2-D photo-detector of the camera represents a coded linear combination in the Fourier domain of all the rays entering the camera from multiple directions. The known linear combination superimposed by the non-refractive mask light field can be decoded by software to obtain the 4-D light field. In general the performance of this radiance camera is similar in terms the spatial and directional resolution it can achieve using a given photo-detector size, in terms of number of pixels, except that the radiance analysis camera may offer increased spatial resolution per view, but the number of views that can be resolved is highly dependent on the computational throughput one is willing to allocate to the post-capture processing. In other words, the improvement in the spatial resolution per view that may be offered by the radiance camera would be at the expense of increased computational resources. Furthermore, the mask used in the radiance camera will cause light loss that would tend to reduce the capture image signal to noise ratio (SNR). In addition, similar to the camera array and the plenoptic camera, for the radiance camera to cover a reasonable angular extent, it must employ as large as possible diameter objective lens which in turn requires a large optical track length, thus making the radiance analysis camera also bulky and not at suitable for embedding in mobile devices.
In general, prior art light field cameras illustrated in FIGS. 1A, 1B and 1C are limited in their functionality and applications because:
1. The depth of their light field is limited by the focus depth of their objective lens;
2. The field of view of their light field is limited by the angular extent of their objective lens;
3. Their objective lens and MLA (micro lens arrays) must have a matched F#, which results in complicated and costly lens system designs;
4. The large diameter of the objective lens needed to achieve a reasonable size field of view typically results in a rather large optical track length which in turn causes the volumetric size of the light field camera to become large, thus reducing the utility of the camera and preventing its use in mobile applications;
5. The objective lens system adds well known optical distortions and aberrations, such as barrel distortion, TV distortion, etc. . . . , which reduce the optical quality of the captured light field and in turn distort the depth and directional information captured by such cameras; and
6. The light field captured by such cameras usually suffers from under-sampling and resultant sampling artifacts because the limited resolution of the sensor, which typically has to be apportioned between the achievable spatial and angular resolution, limits the total number of directions these light field cameras can capture.
It is therefore an objective of this invention to introduce a spatio-temporal light field camera that overcomes the limitations and weaknesses of the prior art, thus making it feasible to create a light field camera that can be embedded in mobile devices and offer the users of such devices the capability of computational focusing of 2D images and the capture of 3D images over a wide angular extent. Additional objectives and advantages of this invention will become apparent from the following detailed description of a preferred embodiment thereof that proceeds with reference to the accompanying drawings.